End to End Alignment Learning of Instructional Videos with Spatiotemporal Hybrid Encoding and Decoding Space Reduction

We solve the problem of how to densely align actions in videos at frame level, with only order occurring available, save time-consuming efforts accurately annotate temporal boundaries each action. propose three task-specific innovations under this scenario: (1) To encode fine-grained spatiotemporal local features and long-range patterns simultaneously, we test popular backbones compare their accuracy training times: (i) a recurrent LSTM; (ii) fully convolutional model; (iii) recently proposed Transformer model. (2) address absence ground truth frame-by-frame labels during training, apply connectionist classification (CTC) on top encoder recursively collect all theoretically valid alignments, further weight these alignments frame-wise visual similarities, avoid significant number degenerated paths improve both recognition computation efficiency. (3) quantitatively assess quality learned alignment, comprehensive set frame-level, segment-level, video-level evaluation measurements. Extensive evaluations verify effectiveness our proposal, performance comparable that supervised approaches across four benchmarks different difficulty data scale.